Mortality in Iraq.

In their Article on mortality before and after the 2003 invasion of Iraq (Nov 20, p 1857), Les Roberts and colleagues use several questionable sampling techniques that should have been more thoroughly examined before publication. Although sampling of 988 households randomly selected from a list of all households in a country would be routinely acceptable for a survey, this was far from the method actually used—a point basically lost in the news releases such a report inevitably engenders. The survey actually only included 33 randomised selections, with 30 households interviewed surrounding each selected cluster point. Again, this technique would be adequate for rough estimates of variables expected to be fairly homogeneous within a geographic region, such as political opinion or even natural mortality, but it is wholly inadequate for variables (such as violent death) that can be expected to show extreme local variation within each geographic region. In such a situation, multiple random sample points are required within each geographic region, not one per 739 000 individuals. In my opinion, such a flaw by itself is fatal, and should have precluded publication in a peer-reviewed journal. However, the authors’ sampling technique is also questionable in other ways. When a town or village was selected from the “cumulative population lists for the Governorate”, the survey team then “drove to the edges of the area and stored the site coordinates”. In personally deciding what constituted the edges of the grid rectangle for “the area”, the team leader could potentially introduce bias into the selecting of an area to sample. In any case, it seems quite likely that the grid rectangles created by driving around in a war zone were much smaller than the original census tracts used in the “cumulative population lists”. Additionally, there is no way to verify that the 30 closest households were those actually interviewed. It is always easier to interview the more vocal households, and there is some leeway in deciding which doorways are closest. The claimed 99·5% response rate makes it seem highly suspect that the 30 closest doorways were actually those rigorously selected. Lastly, when interviewing a “household” about violent deaths, one is likely to hear reports included from an extended family unit. Individuals might be included from a network of family by marriage, including many dozens if not hundreds of individuals. Although such individuals might have even lived in the household at some point, that does not mean they would have actually been living at that location at the time of the survey if they had not been killed. Such a phenomenon is much more likely when reporting violent deaths (due to the extreme emotional import of such occurrences) than when reporting natural deaths. I think such considerations should have been uncovered during a thorough review by statistical experts (as I assume occurred before electronic publication of this article immediately before a US presidential election). If not precluding publication altogether, these fairly obvious points should have been included in a fair editorial analysis accompanying the article.